build: add opt-in Cython generated-source cache - #2933
Merged
Merged
Conversation
Contributor
github-actions
Bot
added
cuda.bindings
juenglin
added
CI/CD
Contributor
Author
|
/ok to test 41c8db9 |
juenglin
force-pushed
the
opt-in-cython-cache
branch
from
September 22, 2026 21:54
41c8db9 to
c929252
Compare
Contributor
Author
|
/ok to test c929252 |
juenglin
marked this pull request as ready for review
September 22, 2026 21:56
This comment has been minimized.
This comment has been minimized.
rwgk
approved these changes
Sep 22, 2026
rwgk
left a comment
rwgk
left a comment
Contributor
There was a problem hiding this comment.
Approving, based on several stages of reviewing with codex gpt-5.6-sol medium. There are no findings anymore.
Add an opt-in Cython cache behind CUDA_PYTHON_CYTHON_CACHE_DIR for cuda.bindings and cuda.core builds. When unset, cythonize() is called without cache= and builds are unchanged. POSIX only; Windows returns None with a warning. _cython_cache_path namespaces the cache by package, Python version, and a SHA-256 of output-affecting config (compiler_directives, compile_time_env, language_level, cplus, debug, cuda_major). This works around cython/cython#7532 (Cython omits compiler_directives from its native fingerprint); the helper and its tests can be removed once that issue is fixed in a released Cython version covered by cuda-python's minimum. _stable_cython_alias creates package-local directory symlinks (.cython-stdlib, and .cython-bindings for cuda.core) before cythonize() and removes them in finally, giving Cython stable relative include paths across PEP 517 builds that install deps under randomized temp prefixes. The two helpers are vendored in both build_hooks.py files (PEP 517 isolation forbids a shared import) and kept in sync by the existing pre-commit hook toolshed/check_build_hooks_sync.py, now covering one merged "shared build helpers" block instead of the toolchain-only block. No runtime drift test. Tests live in cuda_python_test_helpers/cython_cache.py (shared mixins + miss/hit smoke test + cross-isolated-env cache-hit regression) and are exercised by both packages' tests/test_build_hooks.py. POSIX-only tests skip on Windows; a Windows-only test asserts the set-env warn+None path.
juenglin
force-pushed
the
opt-in-cython-cache
branch
from
September 29, 2026 17:13
c929252 to
ba7e14f
Compare
Contributor
Author
|
/ok to test ba7e14f |
juenglin
enabled auto-merge (squash)
September 29, 2026 18:19
Contributor
|
leofang
added a commit
to leofang/cuda-python
that referenced
this pull request
Sep 29, 2026
Resolves conflicts with NVIDIA#2933 (Cython generated-source cache). Ralf's _cython_cache_path and _stable_cython_alias helpers were vendored in both build_hooks.py files behind the # --- begin/end shared build helpers markers his PR renamed. Moved both into cuda_bindings/_build_shared.py alongside the toolchain block; both build_hooks.py now import them via `from _build_shared import ...` (no vendoring, no sync check). Conflict resolution: - cuda_bindings/build_hooks.py, cuda_core/build_hooks.py: keep the from-_build_shared import structure; extend the import list with _cython_cache_path and _stable_cython_alias. - cuda_bindings/_build_shared.py: add the two helpers plus the imports they need (contextlib, hashlib, uuid, warn). - cuda_bindings/build_hooks.py: drop hashlib (moved), keep contextlib (still used for contextlib.suppress), re-add Path from pathlib for the Cython.__file__ path handling in _build_cuda_bindings. - cuda_core/build_hooks.py: drop contextlib, hashlib, uuid, warn (all moved). - .pre-commit-config.yaml: keep my removal of the check-build-hooks-sync entry (script is deleted). - toolshed/check_build_hooks_sync.py: keep my deletion (single source of truth needs no drift check). - Auto-merged test files needed no manual touch; both suites now include Ralf's cython_cache tests and pass (33 bindings, 71 core). Verified after resolution: - ruff check + format: clean. - SPDX: clean. - Sdists for both packages contain the resolved 8579-byte _build_shared.py matching the canonical.
7 of 8 tasks
Andy-Jost
added a commit
to Andy-Jost/cuda-python
that referenced
this pull request
Sep 30, 2026
…untime-floor Conflicts: the toolchain and Cython-cache helpers from NVIDIA#2903 and NVIDIA#2933 landed next to this branch's floor and header-check helpers in both build_hooks.py files and their tests. Both sets are kept. cuda.core's build now stamps the build configuration (main) and checks the cuda-bindings floor and header (this branch) at the same point; the cuda-bindings tests file merges main's toolchain tests with this branch's header-check tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on top of #2903 (
toolchain-override-backend). Merge after #2903.Summary
Adds an opt-in Cython generated-source cache behind
CUDA_PYTHON_CYTHON_CACHE_DIRforcuda.bindingsandcuda.corebuilds. When unset,cythonize()is called withoutcache=and builds are unchanged. POSIX only; on Windows the helper returnsNonewith a warning.Two problems it works around:
_cython_cache_path): Cython's native fingerprint omitscompiler_directives([BUG] Cache fingerprinting ignorescompiler_directivescython/cython#7532). The helper namespaces the cache by package, Python version, and a SHA-256 of output-affecting config (compiler_directives,compile_time_env,language_level,cplus,debug,cuda_major). The helper and its workaround-specific tests can be removed once #7532 is fixed in a released Cython version covered by cuda-python's minimum._stable_cython_alias): Cython hashes the absolute path of each resolved.pxd, and PEP 517 installs land under randomized temp prefixes. Atomic package-local directory symlinks (.cython-stdlib, and.cython-bindingsfor cuda.core) give Cython stable relative include paths; created beforecythonize()and removed infinally.Sync mechanism
The two helpers are vendored in both
build_hooks.pyfiles (PEP 517 isolation forbids a shared import). Drift is enforced by the existing pre-commit hooktoolshed/check_build_hooks_sync.py, which now checks one merged "shared build helpers" block (toolchain + cache helpers) instead of the toolchain-only block. There is no runtime drift test.Tests
Shared tests live in
cuda_python_test_helpers/cuda_python_test_helpers/cython_cache.py(mixins, miss/hit smoke test, cross-isolated-env cache-hit regression) and are exercised by both packages'tests/test_build_hooks.py. POSIX-only tests skip on Windows; a Windows-only test asserts the set-env warn +Nonepath.Notes
cuda.corekeeps its existing per-configurationbuild_dir(cu{major}-{toolchain}-{debug|opt}[-cov]); onlycache=/include_pathwere added.CUDA_PYTHON_CYTHON_CACHE_DIRis intentionally not documented inenvironment_variables.rst; it joinsCUDA_PYTHON_TOOLCHAINunder the "no support guarantee" comment in bothbuild_hooks.pyfiles.